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ABSTRACT 


The  problem  addressed  in  this  research  is  the  need  for  supervisory  or  system 
summary  displays  for  the  Advanced  Tomahawk  Weapons  Control  System  (ATWCS). 
These  displays  are  needed  to  accurately  depict  the  current  system  state  and  weapon  status 
in  order  to  aid  strike  supervisory  personnel  in  making  correct  and  timely  decisions.  This 
research  examined  the  problem  in  the  context  of  designing  a  set  of  graphical  displays  that 
extracts  information  relevant  to  the  strike  supervisor  from  ATWCS  and  displays  it  in  a 
manner  that  allows  both  rapid  and  accurate  interpretation. 

The  approach  used  to  solve  the  problem  progressed  in  four  distinct  phases.  The 
first  phase,  Requirements  Analysis,  consisted  of  gathering  system  requirements  through 
interviews  with  U  S.  Navy  officers  who  have  experience  as  strike  warfare  supervisors.  In 
the  second  phase,  an  initial  design  was  produced  using  Century  Computing’s  rapid 
prototyping  tool  TAB  Plus  Workbench™.  The  third  phase  involved  the  heuristic  and 
guideline  evaluation  of  the  prototype  based  on  accepted  user  interface  design  principles 
and  ATWCS  user  interface  requirement  specifications.  This  evaluation  produced  a  second 
iteration  prototype  that  was  used  in  the  final  phase.  Usability  Testing.  The  prototype  was 
tested  by  U  S.  Navy  Officers  with  Tomahawk  strike  experience  and  test  results  were 
recorded.  Changes  were  then  made  to  the  prototype  to  correct  usability  problems 
discovered  by  the  user  testing,  yielding  a  third  iteration  prototype. 

The  final  result  of  this  research  is  a  set  of  prototype  displays,  in  both  paper  and 
TAE  Plus  Workbench™  resource  file  formats,  that  will  be  provided  to  Naval  Command, 
Control,  and  Ocean  Surveillance  Center  (NCCOSC)  Research,  Development,  Test  and 
Evaluation  Division  (NRaD)  for  consideration  during  system  design  and  implementation. 
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1.  INTRODUCTION 


A.  BACKGROUND 

Of  the  124  surface  combatant  ships  currently  active  in  the  U  S.  Navy,  63  are 
equipped  to  launch  Tomahawk  cruise  missiles  (Sharp,  1995).  Of  the  ten  ships  that  are 
planned  for  construction  through  Fiscal  Year  1996,  eight  will  be  equipped  to  launch 
Tomahawk  cruise  missiles  (Sharp,  1995).  From  these  statistics  one  can  see  that 
Tomahawk  cruise  missiles  do  and  will  play  a  large  role  in  the  U  S,  Navy’s  operations. 

Because  Tomahawk  missiles  have  the  ability  to  strike  a  wide  variety  of  targets 
hundreds  of  miles  inland,  they  provide  operational  commanders  with  many  options. 
Tomahawk  cruise  missiles  can  destroy  targets  that  jeopardize  flight  crews,  such  as 
surface-to-air  missile  sites.  During  contingency  operations,  they  provide  a  rapid  reaction 
capability  in  response  to  a  quick  change  in  hostilities.  Tomahawk  is  a  dependable  weapon 
that  can  be  used  to  strike  a  wide  range  of  targets  giving  operational  commanders  a 
versatile  tool  with  which  to  conduct  wartime  operations. 

Tomahawk  land  attack  cruise  missiles  (TLAMs)  are  complex  missiles  with  equally 
complex  firing  procedures.  Because  of  this  complexity,  strike  watch  teams  typically  have 
five  members:  A  Database  Manager  (DBM),  Engagement  Planner  (EP),  two  Launch 
Controllers  (LCs)  and  an  Engagement  Control  Officer  (ECO),  also  called  the  Surface 
Strike  Warfare  Coordinator  (SSTWC)  on  some  ships.  All  watch  team  members,  except 
the  ECO,  are  seated  at  an  Advanced  Tomahawk  Weapon  Control  System  (ATWCS) 
console  at  which  they  carry  out  the  tasks  associated  with  their  position,  ATWCS  is  the 
shipboard  system  that  encapsulates  all  the  control  functions  necessary  to  launch  TLAMs. 

The  ECO,  acting  as  the  overall  strike  supervisor,  provides  guidance  to  each  of  the 
strike  team  watchstanders  based  on  current  mission  requirements  and  the  tactical  situation. 
The  ECO  also  advises  the  Commanding  Officer  (CO),  Tactical  Action  Officer  (TAO)  and 
Officer  of  the  Deck  (OOD)  on  all  matters  concerning  a  TLAM  strike.  The  ECO’s  duties 
include  keeping  the  CO  and  TAO  informed  of  the  mission  status.  This  consists  of 
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information  such  as  any  new  TLAM  mission  tasking  received,  the  progress  of  TLAM 
missile  warm-up,  the  current  operational  status  of  the  system,  etc.  The  ECO  also  keeps 
the  OOD  informed  of  launch  positions  and  launch  times. 

To  effectively  perform  his  duties  and  make  proper  decisions,  the  ECO  needs 
access  to  a  variety  of  information  available  from  the  weapon  control  system.  Currently, 
there  is  no  single  resource  in  ATWCS  that  provides  this  information.  Instead,  the 
information  is  scattered  throughout  the  various  subsystems;  database  management, 
engagement  planning  and  launch  control.  As  a  result,  the  ECO  must  frequently  interrupt 
console  operator’s  tasks  to  receive  updates  to  time  critical  information. 

To  further  complicate  matters.  Tomahawk  mission  tasking  is  usually  quite  intense, 
requiring  the  ECO  to  track  multiple  engagements,  each  with  multiple  missiles, 
simultaneously.  This  fact  is  supported  by  historical  data  from  Operation  Desert  Storm, 
the  first  use  of  Tomahawk  cruise  missiles  in  combat  (Office  of  the  CNO,  1991).  Cohen 
(1993)  shows  that  during  the  first  day  of  hostilities,  122  Tomahawk  land  attack  missiles 
(TLAMs)  were  fired  with  a  steady  decline  in  tasking  throughout  the  first  week.  In  all,  288 
TEAMS  were  fired  from  just  16  ships  and  two  submarines  with  the  US  S  FIFE  (DD  991) 
launching  58  missiles  during  the  course  of  the  war  (Office  of  the  CNO,  1991).  The 
success  of  TLAM  in  Desert  Storm  has  made  it  the  weapon  of  choice  for  disrupting  enemy 
operations  and  hitting  targets  when  aircraft  are  either  not  available  or  unable  to  fly  due  to 
heavy  anti-air  defenses.  More  recently,  the  US  S  NORMANDY  (CG  61)  launched  13 
TLAMs  into  Bosnia  on  September  10,  1995  to  destroy  an  air  defense  site  (Surface 
Warfare,  1995).  Based  on  this  historical  data,  it  is  a  reasonable  assumption  that  future 
conflicts  will  see  the  extensive  use  of  TLAM  early  in  the  hostilities.  Such  tasking  will 
require  ECOs  to  keep  track  of  multiple  missions  whose  launches  must  be  timed  to  the 
second  to  meet  strike  objectives. 

The  aim  of  this  research  is  to  provide  the  ECO  with  a  tool  to  reduce  the  task  load 
associated  with  supervising  a  TLAM  strike.  This  will  be  done  by  using  usability 
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engineering  methods  to  design  graphical  displays  that  give  the  ECO  a  single  source  of 
necessary  information.  These  summary  displays  will  be  located  on  a  small  LCD  video 
monitor  attached  to  the  top  of  the  standard  ATWCS  monitor. 

B.  OBJECTIVES 

The  purpose  of  this  thesis  is  to  present  a  series  of  prototype  graphical  displays  for 
ATWCS  that  are  tailored  for  use  by  strike  team  supervisors.  This  research  employs 
usability  engineering  methods  to  determine  what  information  strike  supervisors  typically 
need  throughout  the  course  of  an  engagement  and  the  best  way  to  display  it.  The  product 
of  this  research  is  a  set  of  graphical  displays  that  will  allow  rapid  and  accurate  assimilation 
of  system  data  by  strike  supervisors. 

There  are  several  objectives  to  reach  before  achieving  the  final  goal  of  a  set  of 
prototype  displays.  The  first  objective  is  to  determine  exactly  what  information  the  ECO 
needs  to  perform  his  duties.  This  information  is  analyzed  to  determine  the  level  of  detail 
required  by  the  ECO.  Further,  the  information  is  prioritized  to  better  determine  how  it  is 
to  be  presented  to  the  ECO.  The  second  objective  is  the  production  of  a  first  iteration 
prototype  that  not  only  displays  the  information  accurately,  but  allows  for  rapid 
assimilation.  This  prototype  is  designed  within  the  established  framework  of  JCM-2154, 
the  ATWCS  Human-Computer  Interface  Requirements  Specification,  and  conforms  to 
general  interface  design  guidelines.  As  a  fiarther  requirement,  the  displays  must  fit  on  the 
size-limited  secondary  screen.  The  final,  and  perhaps  most  important,  objective  is  that  the 
summary  displays  be  simple  to  use  and  understand.  This  objective  is  met  using  both 
heuristic  evaluation  and  usability  testing  with  experienced  ECOs  to  reveal  usability 
problems  and  make  corrections  to  the  prototype. 

C.  INTERFACE  ENVIRONMENT 

1.  Hardware  Environment 

The  summary  displays,  hereafter  called  ECO  displays,  presented  in  this  thesis  must 
fit  on  the  limited  screen  space  of  the  ATWCS  console’s  secondary  LCD  screen.  The 
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overall  resolution  of  the  screen  is  640  pixels  by  480  pixels;  however,  required  interface 
elements  reduce  the  usable  screen  space  to  631  by  395  pixels.  The  ECO  display’s  window 
frame  further  reduces  the  application  area  to  61 1  by  356  pixels.  Figure  1  shows  the  layout 
of  the  secondary  screen  along  with  its  associated  dimensions. 


Overall  window.  Including  frame.  (631  x  395  pixels) 


Figure  1 .  Secondary  Screen  Layout. 


JCM-2154  states  that  supervisory  information  should  appear  on  the  upper  display. 
Although  the  secondary  screen’s  usable  application  area  is  very  small,  its  physical  location 
makes  it  the  ideal  place  for  displays  intended  for  ECOs.  Due  to  the  nature  of  the  ECO’s 
duties  and  the  lack  of  a  dedicated  console  for  his  use,  the  ECO  typically  remains  standing 
behind  the  watch  team  members  throughout  the  strike.  The  elevated  position  of  the 
secondary  screen  allows  information  to  be  viewed  by  the  ECO  without  having  to  lean  over 
and  interrupt  a  watchteam  member. 

2.  Software  Environment 

The  ECO  displays  are  designed  as  an  addition  to  an  existing  system.  As  such,  they 
must  strictly  adhere  to  the  established  HCI  guidelines  and  requirements  for  the  parent 
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system.  JCM-2154  is  the  governing  document  for  all  ATWCS  HCI  requirements.  It 
defines  both  the  appearance  and  the  behavior  of  the  interface  for  ATWCS. 

ATWCS  uses  the  UNIX  operating  system,  X  Window  System  window  manager 
and  the  Open  Software  Foundation  (OSF)  Motif  graphical  user  interface  widget  set.  The 
X  Window  System,  or  X,  provides  functions  for  creating  and  managing  windows  as  well 
as  for  drawing  in  these  windows.  Motif  is  a  set  of  preconstructed  user  interface  elements, 
called  widgets,  that  provides  a  high  level  application  interface  with  X.  Motif  provides  a 
common  look  and  feel  to  application  programs  that  use  it,  providing  consistency  between 
the  interfaces  of  different  applications  (Brain,  1992).  In  other  words,  a  Motif  button  in 
one  application  looks  and  behaves  just  like  a  Motif  button  in  another.  JCM-2154  further 
defines  how  each  Motif  interface  element  should  appear  and  behave  within  the  framework 
of  ATWCS. 

D.  INTERFACE  DESIGN  METHODOLOGY 

The  methodology  used  to  design  the  ECO  displays  is  adapted  from  the  usability 
life  cycle  presented  in  Nielsen  (1993).  Since  the  displays  were  designed  as  an  addition  to 
an  existing  system,  many  steps  of  the  life  cycle  had  already  been  performed  by  the  parent 
system’s  developers,  resulting  in  JCM-2154,  a  very  detailed  HCI  requirements 
specification.  The  remaining  steps  that  were  needed  to  design  the  ECO  displays  were 
carried  out  in  four  phases:  requirements  analysis,  design  and  prototyping,  heuristic  and 
guideline  evaluation  and  usability  testing. 

The  methodology  used  results  in  an  iterative  design  process  that  is  focused  on  the 
needs  of  the  user.  The  first  two  phases  extract  the  user’s  requirements  and  transform 
them  into  an  initial  prototype  designed  to  conform  to  existing  interface  requirements. 
Well-defined  sets  of  heuristics  and  guidelines  ensure  the  interface  conforms  with  sound 
interface  design  principles  to  reveal  obvious  usability  problems.  Usability  testing  subjects 
the  prototype  to  a  final  evaluation  conducted  by  the  user.  Problems  found  in  testing  are 
corrected  to  yield  a  third  iteration  prototype. 
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E.  OUTLINE 

Each  of  the  next  four  chapters  of  this  thesis  present  a  different  phase  of  the 
methodology.  The  issues,  detailed  methods  and  results  of  each  phase  are  presented.  At 
the  end  of  Chapters  III,  IV  and  V,  the  resulting  interface  design  will  be  presented.  The 
final  chapter  will  include  conclusions  and  recommendations  about  both  the  design  method 
used  and  the  resulting  product. 
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11.  REQUIREMENTS  ANALYSIS 


The  goals  of  requirements  analysis  are  to  define  the  purpose  of  a  proposed 
software  system  and  determine  any  constraints  on  its  development  (Berzins  and  Luqi, 
1991).  When  designing  a  user  interface,  requirements  analysis  also  entails  determining  the 
user  tasks  that  the  interface  must  support,  User  tasks  and  needs  are  defined  in  terms  of 
the  users  and  their  environment,  with  no  reference  to  how  the  system  will  support  those 
tasks  or  meet  those  needs  (Lewis  and  Rieman,  1993).  Simply  put,  requirements  analysis 
specifies  what  a  piece  of  software  is  supposed  to  do,  not  how  it  is  supposed  to  do  it. 

This  chapter  specifies  the  requirements  for  the  Engagement  Control  Officer  (ECO) 
displays.  These  requirements  include  what  information  should  be  on  the  displays  and  any 
constraints  on  the  displays.  The  informational  requirements  are  divided  based  on  a 
taxonomy  that  considers  each  piece  of  information’s  importance  to  the  ECO  during  the 
course  of  a  Tomahawk  Land  Attack  Missile  (TEAM)  strike. 

A.  REQUIREMENTS  SOLICITATION 

To  best  determine  what -information  would  benefit  an  ECO  during  a  strike,  15  U  S. 
Navy  Surface  Warfare  Officers  were  questioned.  Each  officer  has  ECO  and  Tactical 
Action  Officer  (TAO)  experience  from  different  Tomahawk  Land  Attack  Missile 
(TLAM)-capable  platforms.  During  initial  contact,  the  concept  of  the  ECO  displays  was 
explained.  Because  ATWCS  is  not  yet  on  board  most  TLAM-capable  ships,  interviewees 
were  given  a  description  of  the  hardware  for  the  ATWCS  console.  Each  officer  was  then 
asked  to  list  the  information  they  required  during  a  TLAM  strike;  particularly,  any 
information  that  requires  interrupting  a  console  operator  to  obtain.  Subsequent 
communication  involved  questioning  the  interviewees  about  their  replies  to  further  narrow 
their  specific  requirements. 

All  but  one  interviewee  relied  on  recent  experience  with  TLAM  operations  to 
determine  the  information  they  would  find  useful.  The  remaining  officer,  currently  serving 
as  Strike  Warfare  Officer  on  board  a  destroyer,  conducted  several  strike  simulations  in 
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order  to  determine  the  information  he  would  find  helpfial.  During  these  simulations  he 
noted  the  information  that  he  continually  required  and  that  which  his  chain  of  command 
requested. 

To  ensure  the  officers  not  assigned  to  ships  accurately  recalled  the  information 
needed  during  a  TLAM  strike,  their  requirements  were  compared  to  those  found  during 
the  simulations.  Since  the  requirements  from  both  sources  coincide,  those  based  on 
memory  alone  are  considered  accurate.  Also,  as  all  of  the  requirements  were  listed  by  at 
least  two  officers,  the  requirements  are  the  information  required  during  a  TLAM  strike. 

B.  REQUIREMENT  TAXONOMY 

To  better  decide  how  to  display  the  information  on  the  ECO  displays,  the  notion 
that  one  piece  of  information  can  be  more  important  than  another  must  be  introduced.  If 
the  informational  requirements  are  not  prioritized  there  is  a  chance  that  a  critical  piece  of 
information  might  not  be  displayed  prominently  for  the  ECO.  Likewise,  a  piece  of 
information  that  is  not  critical  might  distract  the  ECO  due  to  its  placement,  text  size,  etc. 
Each  requirement  is  placed  into  one  of  three  categories  having  the  following 

criteria: 

•  Critical  -  Constantly  or  frequently  updated  information  that  the  ECO  needs 
repeatedly.  Critical  information  should  be  visible  at  all  times. 

•  Summary  -  The  information  in  this  category,  when  viewed  together,  depicts  the 
overall  system  status.  Summary  information  should  be  displayed  together  on  a 
single  screen. 

•  Detail  -  This  information  provides  the  details  needed  by  the  ECO  to  perform 
his  duties.  Detail  information  can  be  spread  out  over  several  screens  but 
should  be  grouped  together  with  similar  information  where  possible. 

C.  REQUIREMENTS 

1.  Informational  Requirements 

Figure  2  lists  the  informational  requirements  resulting  from  requirements  elicitation 
and  divides  them  according  to  the  taxonomy  discussed  previously.  The  requirements  are 
divided  based  on  comments  from  the  interviewees. 
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CRITICAL 


•  Time  Until  Launch  (TUL)  of  the  next  missile  to  be  launched 

Course  and  speed  to  arrive  at  launch  point  of  next  missile  to  be  launched  at  its 
Time  of  Launch  (TOL) 

•  The  status  of  Officer  In  Tactical  Command  Information  Exchange  System 
(OTCIXS)  and  the  time  since  data  was  last  received  (time  late). 

•  System  mode  (Training/Tactical) 

•  Overall  Vertical  Launching  System  (VLS)  status  (Fault/No  Fault) 

•  Bearing,  range  and  ATWCS  track  number  of  nearest  Critical  Contact  of 
Interest  (CCOI) 

SUMMARY 

•  Maximum  salvo  available  of  each  TLAM  variant 

•  For  the  next  plan  to  be  launched: 

o  Plan  number 
o  Plan  status 
o  TUL 

o  Time  on  Target  (TOT) 
o  TOL 

o  Missile  alignment  status  of  the  last  missile 
o  Plan  Salvo 

o  Review  Required  (Yes/No) 

•  Launch  Control  Unit  (LCU)  configuration  and  status 

•  Status  of  Command  and  Decision  (C&D)  interface 

•  Status  of  Inertial  Navigation  System  (INS)  and  its  alignment  (Forward/Aft) 

•  Launchers  authorized  and  selected 

•  VLSs  authorized 

•  VLS  controlling  Weapon  Control  System  (WCS) 

•  VLS  mode  (Tactical/Simulated) 

•  VLS  State 

•  VLS  inventory  status 

•  VLS  Launch  Enabled 

•  Engagement  Planning  Console  (EPC)  mode 

•  Launch  Control  (LC)  mode 


Figure  2.  Informational  Requirements  for  ECO  Displays. 


9 


I  DETAIL 

i 

! 

I  •  For  each  active  TLAM  plan; 

j  o  Plan  Number 

o  Plan  Status 
o  Plan  Order 
o  Time  of  Launch  (TOL) 
o  TUL 

0  Time  on  Target  (TOT) 

o  Missile  alignment  status  of  last  missile 

o  Plan  Salvo,  Ready  Spare  and  Backup  (#  of  missiles) 

o  Number  of  missiles  selected 

o  Review  Required  (Yes/No) 

o  True  and  Relative  Bearing  of  missile  departure 

o  In  launch  basket?  (Yes/No) 

o  Bearing,  and  range  to  launch  point 

o  Required  course  and  speed  to  arrive  at  launch  point  at  TOL 

•  For  each  missile  assigned  to  a  plan: 

o  Cell  number 
o  Missile  type 
o  Associate  plan  number 
o  Launch  Side  (Port/Starboard) 
o  Alignment  status 
o  Booster  status 
o  Missile  status 
o  tOL 

•  For  each  VLS: 

o  Missile  type  in  each  cell 
o  Plan  number  of  cells/missiles  assigned  to  a  plan 
o  Any  modules  with  faults 
o  Location  of  any  inoperable  missiles 
o  Location  of  missiles  being  aligned 
o  Location  of  missiles  that  have  completed  alignment 

•  Bearing,  range,  course  and  speed  of  all  COIs/CCOIs 


(Figure  2  continued.)  Informational  Requirements  for  ECO  Displays. 
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2.  Interface  Requirements  and  Constraints 

The  ECO  Displays  must  conform  to  the  requirements  listed  in  JCM-2154. 
Appendix  B  of  JCM-2154  contains  a  checklist  that  can  be  used  to  ensure  compliance  with 
the  requirements  specification.  In  addition  to  the  requirements  listed  in  that  appendix,  the 
interface  is  constrained  by  the  following  physical  characteristics: 

•  The  displays  must  fit  in  the  secondary  LCD  screen. 

•  The  overall  window  size  shall  not  exceed  631  by  395  pixels,  including  window 
frame. 

•  The  application  area  of  the  window  shall  not  exceed  61 1  by  356  pixels. 

•  The  ECO  displays  shall  use  a  viewing  distance  of  26  inches.  Based  on  JCM- 

2 1 54  requirements  the  font  shall  be  Helvetica  with  a  minimum  of  13  point  bold 
and  a  maximum  of  24  point  with  a  preferred  size  of  1 8  points. 

D.  CONCLUSION 

This  chapter  lists  the  requirements  that  fleet  ECOs  need  to  conduct  a  TLAM 
strike.  These  user  requirements  drive  the  design  of  the  ECO  displays.  The  displays  must 
incorporate  all  the  requirements  in  order  to  meet  the  needs  of  the  user.  The  design  must 
also  take  into  account  the  requirement  taxonomy  and  the  limited  display  area  of  the 
secondary  screen. 
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in.  INTERFACE  DESIGN 


A.  DESIGN  CONSIDERATIONS 

As  with  any  interface,  there  are  many  issues  influencing  the  design  of  the 

Engagement  Control  Officer  (ECO)  displays.  When  designing  a  user  interface,  the 

designer  must  always  keep  in  mind  the  function  of  the  interface.  The  function  of  the  ECO 

displays  is  to  display  required  information  to  the  ECO  during  a  Tomahawk  Land  Attack 

Missile  (TEAM)  strike.  This  information  must  be  displayed  in  such  a  way  that  the  ECO 

can  interpret  it  accurately  and  quickly. 

If  the  system  does  not  present  the  information  that  the  user  needs,  or 
presents  it  in  an  unusable  or  confusing  manner,  the  user  may  decide  not  to 
use  the  system...,  or  the  user’s  ability  to  perform  the  necessary  task  may  be 
sharply  degraded  (Tullis,  1988). 

The  small  size  of  the  secondary  screen  is  also  a  primary  consideration  in  the  design 
of  the  ECO  displays.  All  the  required  information  must  fit  in  the  limited  space  allowed  on 
the  secondary  screen.  This  requires  that  information  be  displayed  as  tersely  as  possible 
without  a  loss  of  information.  The  information  must  be  both  concise  and  complete  to 
prevent  inaccurate  interpretation  by  the  ECO. 

Mayhew  (1992)  states  “user  interface  design  is  a  matter  of  compromise  and  trade¬ 
off.”  Often  the  goals  of  accurate  and  rapid  assimilation  and  minimum  screen  use  seem 
mutually  exclusive.  In  such  cases,  a  compromise  must  be  reached  that  allows  the  most 
complete  data  in  the  smallest  screen  space  possible. 

The  ATWCS  Human  Computer  Interface  (HCI)  Requirements  Specification,  JCM- 
2154,  clearly  defines  the  framework  to  which  the  ECO  displays  must  conform.  This 
interface  framework  must  also  be  considered  when  designing  the  ECO  displays. 

Although  designing  an  interface  under  such  strict  standards  may  seem  restrictive,  it 
provides  several  advantages.  Interface  standards  provide  guidelines  for  the  operation  and 
visual  presentation  of  interface  elements  ensuring  that  a  button  “looks  and  feels”  the  same 
throughout  an  application.  These  same  interface  elements  provide  standard  methods  of 
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interaction.  (Lewis  and  Rieman,  1994)  For  example,  to  select  an  option  from  a  list  of 
mutually  exclusive  list  is  often  done  using  a  set  of  radio  buttons  (Open  Software 
Foundation,  1993).  Designing  to  meet  an  existing  standard  allows  the  designer  to 
concentrate  on  the  best  way  to  present  the  interface  rather  than  the  details  of  how 
interface  elements  must  interact  with  the  user  (Lewis  and  Rieman,  1994). 

B.  PROTOTYPING 

According  to  Nielsen  (1993),  the  idea  behind  prototyping  is  to  quickly  and  cheaply 
develop  something  that  can  be  tested  with  real  users.  In  the  case  of  a  user  interface, 
prototypes  allow  interactive  user  testing  to  find  usability  problems  before  implementation. 
Typically,  an  interface  prototype  is  tested  and  modified  iteratively  as  usability  problems  are 
uncovered  and  corrected.  This  strategy  is  effective  because  it  is  normally  less  expensive 
and  time  consuming  to  correct  problems  during  the  prototyping  phase  than  after  a  system 
has  been  implemented. 

To  save  cost  and  time,  prototypes  are  a  scaled  down  versions  of  the  final  system, 
lacking  either  features  or  functionality  of  the  full  system.  These  two  dimensions  of 
prototyping  are  described  in  Nielsen  (1993)  as  vertical  and  horizontal  prototypes.  A 
vertical  prototype  is  one  which  fully  implements  only  selected  features  of  a  system.  For 
example,  a  vertical  prototype  of  an  address  book  program  might  implement  data  entry  and 
retrieval  with  real  data  but  no  search  capability.  Vertical  prototypes  can  only  be  used  to 
test  portions  of  a  system,  although  this  testing  will  be  under  real  circumstances  with  real 
user  tasks.  A  prototype  that  has  a  reduced  level  of  functionality  is  called  a  horizontal 
prototype.  A  horizontal  prototype  is  a  surface  layer  that  includes  the  entire  user  interface 
but  has  no  underlying  functionality.  (Nielsen,  1993)  A  horizontal  prototype  of  an  address 
book  program  would  include  all  data  entry  screens,  search  dialog  boxes,  etc.,  but  have  no 
ability  to  store  or  retrieve  data. 

Prototypes  can  implemented  in  several  ways.  An  interface  design  could  be 
prototyped  by  using  paper  mock-ups.  A  developer  might  design  the  prototype  of  a  fiiture 
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system  on  a  platform  different  from  the  final  system’s  target  environment.  A  prototype 
could  also  be  constructed  using  a  generalized  scripting  language  such  as  shell  scripts 
rather  than  a  true  programming  language.  There  also  exist  many  tools  designed 
specifically  for  prototyping.  (Nielsen,  1993)  Each  of  these  methods  have  their  oAvn 
advantages  and  disadvantages  which  are  discussed  fully  in  Nielsen  (1993). 

The  ECO  displays  are  a  horizontal  prototype  constructed  with  the  aid  of  TAE  Plus 
Workbench™,  a  rapid  prototyping  tool  from  Century  Computing.  TAE  Plus 
Workbench™  allows  a  developer  to  visually  construct  an  interface  and  then  provide 
limited  functionality  by  using  a  scripting  language.  The  interface  can  then  be  rehearsed  in 
real  time  during  development.  Another  advantage  of  this  tool  is  that  it  can  generate  high 
level  language  code,  such  as  C  or  Ada,  to  implement  the  interface,  shortening 
development  time.  TAE  Plus  Workbench™  does  have  a  disadvantage  in  that  it  does  not 
fully  support  some  functionality  found  in  X- Windows  and  Motif 

TAE  Plus  Workbench™  is  the  development  tool  for  the  ECO  displays  for  several 
reasons.  First,  its  visual  nature  allows  it  to  display  the  application  area  and  interface 
objects  of  the  ECO  displays  in  their  true  size.  This  is  especially  important  because  the 
limited  space  available  to  the  displays  must  be  utilized  effectively.  Second,  TAE  Plus 
Workbench™  allows  the  interface  to  be  rehearsed  interactively.  This  allows  users  to 
actually  see  how  the  ECO  displays  look  and  feel  during  user  testing  without  having  to 
write  application  code.  Finally,  the  tool’s  availability  and  compatibility  with  JCM-2154 
standards  make  it  an  ideal  development  environment. 

C.  OVERALL  INTERFACE  CONCEPT 

Good  interface  design  is  not  a  matter  of  applying  a  set  of  rules  or  algorithms  to 
achieve  a  usable  interface.  As  discussed  in  Mayhew  (1992),  there  are  many  principles  for 
designing  a  good  interface.  However,  just  knowing  these  principles  is  not  enough  to 
ensure  a  usable  interface.  A  good  interface  comes  from  knowing  the  user,  his  tasks  and 
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his  goals.  The  design  of  the  ECO  displays  is  driven  by  the  nature  and  the  volume  of  the 
information  that  it  must  present. 

Obviously,  there  are  many  ways  to  implement  the  ECO  displays,  but  one 
characteristic  will  be  shared  by  any  design.  Because  the  amount  of  information  required 
by  the  ECO  is  quite  large,  it  is  apparent  that  any  design  will  have  to  use  multiple  screens, 
or  pages,  to  fit  the  information  on  the  secondary  screen.  This  requires  a  navigation 
method  to  switch  between  the  display’s  pages. 

The  nature  of  the  information  to  be  displayed  greatly  influences  the  conceptual 
design.  Critical  information  must  be  displayed  at  all  times  implying  that  some  portion  of 
the  ECO  displays  be  constantly  visible.  Summary  information  must  be  grouped  together 
coherently.  This  suggests  that  all  the  summary  information  be  displayed  on  the  same  page 
Detail  information  should  be  grouped  together,  indicating  the  need  for  pages  that  contain 
similar  information. 

To  account  for  the  various  categories  of  information  that  must  be  displayed  the 
following  window  layout  scheme  is  used.  The  basic  window  of  the  displays  is  sized  to  the 
maximum  size  for  the  secondary  screen  of  63 1  by  395  pixels,  making  the  application  area 
611  by  356  pixels.  To  provide  an  area  that  is  constantly  visible,  the  top  45  pixels  of  the 
application  area  will  remain  static.  All  critical  information  will  appear  in  this  permanent 
status  bar.  The  remaining  application  area  below  the  status  bar  will  be  used  for  summary 
and  detail  information.  All  summary  information  will  appear  on  a  single  summary  page, 
while  detail  information  will  be  grouped  together  and  placed  on  a  series  of  several 
displays. 

As  mentioned  previously,  a  navigation  method  is  required  to  view  the  various 
pages  of  the  ECO  display.  Rather  than  provide  a  navigation  mechanism  that  would  use  a 
portion  of  the  application  area,  such  as  tabs,  the  On-Screen  Variable  Action  Buttons 
(OSVs)  already  present  on  the  secondary  screen  are  used.  OSVs  are  push  buttons  whose 
changing  labels  can  simulate  a  menu  hierarchy.  Their  placement  adjacent  to  the  ECO 
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display’s  window  and  the  fact  they  use  no  additional  application  area  make  them  an  ideal 
choice.  The  functionality  of  OS  Vs  is  fully  described  in  JCM-2 1 54. 

D.  DESIGN  DETAILS 

As  discussed  previously,  the  amount  of  information  to  be  presented  requires  the 
ECO  displays  be  divided  into  several  pages.  Based  on  the  nature  of  the  informational 
requirements,  the  displays  are  split  into  the  following  five  pages:  ECO  Summary, 
COI/CCOIs,  Forward  VLS  Status,  Aft  VLS  Status  and  Plan  Status.  At  the  top  of  every 
page  is  a  status  bar  that  contains  all  the  critical  information  listed  in  the  informational 
requirements.  To  navigate  between  these  pages,  the  OS  Vs  are  configured  as  shown  below 
in  Figure  3 . 


ECO 

Summary 

Plan 

Status 

VLS 

Status 

CCOI/COIs 

1  0 
V  L 


Forward 

Aft 

VLS 

VLS 

Back 

Figure  3.  OSV  Configuration. 


Because  the  main  objective  of  the  ECO  displays  is  to  present  information  that  can 
be  interpreted  accurately  and  quickly,  particular  attention  was  paid  to  screen  layout. 
There  are  literally  thousands  of  data-display  and  screen  design  principles  available  to  the 
designer,  such  as  those  in  Schniederman  (1992)  and  Mayhew  (1992).  Many  of  these 
guidelines  are  based  on  the  work  of  Tullis  (1988)  who  reviewed  the  results  of  several 
empirical  studies  of  screen  design.  Based  on  the  observed  results  of  those  studies,  Tullis 
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offers  many  general  guidelines  on  how  best  to  format  screen  displays  for  maximum 
accuracy  in  minimum  time  and  space.  Throughout  the  design  of  the  ECO  displays,  the 
principles  in  Tullis  (1988)  are  to  be  applied  whenever  possible. 

E.  DESIGN  DECISIONS 

1.  Layout  of  Critical  Information  on  the  Status  Bar 
CL  Alternative  Approaches 

The  critical  information  could  be  placed  on  the  status  bar  in  just  about  any 
arrangement.  The  limited  vertical  size  of  the  status  bar  (45  pixels)  prevents  grouping 
information  in  vertical  columns  over  2  rows  high. 
b.  Solution  and  Discussion 

Tullis  (1988)  presents  several  options  for  the  sequence  of  data 
presentation.  The  most  applicable  of  these  in  the  case  of  the  critical  information  is 
sequencing  by  importance.  Another  consideration,  not  included  in  Tullis,  is  the  update 
frequency  of  the  data. 

The  critical  data  can  be  divided  logically  as  listed  below: 

•  Next  launch  data: 
oTUL 

o  Course  and  speed  to  launch  positions 

•  OTCIXS  information: 
o  Status  (up  or  down) 

o  Time  late  of  OTCIXS 

•  VLS  status: 

o  Forward  VLS  status 
o  Aft  VLS  status 

•  Training  Mode 

•  Track  number,  bearing  and  range  of  the  nearest  CCOI 

One  arrangement  is  to  have  four  columns  of  two  rows  each,  with  the  two  single  element 
groups  making  a  column.  Of  these,  the  next  launch  data  information  is  the  most  time 
critical,  therefore  it  should  be  the  first  column.  The  OTCIXS  data  can  be  considered  least 
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critical  in  that  the  loss  of  the  OTCIXS  link  will  not  prevent  a  missile  launch,  consequently, 
it  should  be  in  the  last  column.  Of  the  two  remaining  groups,  neither  one  has  any  reason 
to  precede  the  other,  so  any  arrangement  of  these  two  is  acceptable. 

2.  Navigation  Method  Between  Pages 

a.  Alternative  Approaches 

There  are  several  navigation  methods  that  would  be  effective  for  the  ECO 
displays.  These  include  tabbed  pages,  push  buttons  attached  to  the  status  bar,  a  menu  bar, 
a  pop-up  menu,  hot-keys  and  the  OSVs. 

b.  Solution  and  Discussion 

Of  all  the  alternatives  mentioned  above  only  the  last  three  do  not  require 
application  area  be  taken  away  from  the  ECO  displays.  Because  space  is  so  limited,  the 
first  two  alternatives  are  not  viable  options. 

The  OSVs  are  the  best  of  the  remaining  three  options  for  the  following 

reasons: 


•  Hot  keys,  while  they  normally  offer  short  cuts  for  experts,  require  a  two  handed 
operation  such  as  Control-K.  The  OSVs,  for  all  but  the  VLS  status  pages, 
require  only  a  single  keystroke.  For  the  VLS  status  pages,  two  keystrokes  are 
necessary. 

•  A  pop-up  menu  requires  mouse  interaction  within  the  application  area  of  the 
window.  Because  the  ECO  displays  are  on  the  secondary  screen,  mouse 
interaction  can  be  distracting  to  the  console  operator.  One  of  the  purposes  for 
the  ECO  displays  is  to  prevent  interruption  of  the  console  operator’s  tasks, 
therefore  a  pop-up  menu  is  not  the  best  choice.  If  mouse  interaction  is  desired, 
the  OSVs  can  be  pushed  just  like  any  other  pushbutton. 

•  The  OSVs’  proximity  to  the  ECO  displays  connect  the  two  visually. 

3.  Method  of  Grouping  Information 
0.  Alternative  Approaches 

To  display  many  items  on  the  same  screen,  it  is  helpful  to  organize  them 
into  semantically  related  groups  (Mayhew,  1992).  Because  space  is  so  limited  on  each 
page  of  the  ECO  displays,  this  is  especially  critical.  To  set  off  the  groups  from  one 
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another  there  are  three  options  presented  in  Tullis  (1988):  white  space,  graphical 
boundaries  and  highlighting, 

b.  Solution  and  Discussion 

Because  the  font  used  in  the  ECO  displays  is  Helvetica  14  point  bold  as 
required  by  JCM-2154,  highlighting  becomes  a  difficult  issue.  To  highlight  an  entire 
group  would  mean  that  quite  a  bit  of  information  would  have  to  italicized,  displayed  in 
reverse  video  or  have  the  intensity  changed.  However,  any  of  these  techniques  would  only 
serve  to  clutter  the  display.  Because  of  the  density  of  information  displayed  on  the  various 
pages,  white  space  alone  is  not  sufficient  to  convey  the  visual  concept  of  information 
groups.  Graphical  boundaries  are  the  best  option  for  setting  apart  groups  of  information 
in  the  ECO  displays.  They  enable  the  information  to  be  set  apart  without  cluttering  up  the 
display  or  using  excessive  space.  Although  any  graphical  boundary  might  be  effective,  a 
logical  choice  from  the  programmers  point  of  view  is  the  X-Workspace. 

Labeling  of  each  group  should  be  obvious  and  easily  associated  with  the  group. 

A  label  in  a  large  font  (24  point  bold)  placed  near  the  group  is  sufficient. 

4.  OSV  Hierarchy 

a.  Alternative  Approaches 

The  four  OSVs  must  allow  the  user  to  access  five  pages:  ECO  Summary, 
COI/CCOI,  Forward  VLS  Status,  Aft  VLS  Status  and  Plan/Missile  Status.  The  changing 
labels  on  the  OSVs  allow  almost  any  possible  layout.  Two  possible  alternatives  include: 

•  The  four  OSVs  are  labeled  to  access  the  four  pages  not  currently  displayed. 

When  an  OSV  is  pressed  its  label  changes  to  the  last  page  displayed. 

•  Have  OSVs  labeled  for  ECO  Summary,  COI/CCOI  and  Plan/Missile  Status.  The 
final  OSV  is  labeled  VLS  Status.  Upon  depressing  VLS  Status  all  buttons  are 
cleared  and  two  buttons  are  relabeled  for  Forward  and  Aft  VLS.  The  display  is 
changed  after  the  user  selects  their  desired  launcher. 

b.  Solution  and  Discussion 

The  first  option  discussed  above  is  not  an  acceptable  solution  because  it 
violates  the  idea  of  consistency.  Each  time  an  OSV  is  pressed  the  label  changes  to  the 
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previous  page  displayed.  This  means  that  the  order  of  the  OSVs  will  change  constantly. 
This  prevents  the  user  from  remember  which  keys  correspond  to  which  page. 

The  second  option  is  more  consistent  as  all  the  first  level  button  labels 
always  appear  in  the  same  positions.  However,  it  does  not  provide  any  mechanism  to 
back  out  of  the  second  level  labels  if  the  user  decides  not  to  look  at  a  VLS  launcher  or  hit 
the  incorrect  key. 

A  better  option  is  to  extend  the  second  level  of  the  second  option  discussed 
above.  The  addition  of  a  button  to  return  to  the  previous  level  enables  the  user  to  change 
his  mind  or  back  out  in  the  case  of  a  incorrect  key  press. 

5.  Method  of  Highlighting  Information 

a.  Alternative  Approaches 

There  are  many  methods  to  highlight  information  including  reverse  video, 
color,  brightness  or  boldness,  underlining  and  flashing  (Tullis,  1988).  Another  method  to 
highlight  a  piece  of  information  is  to  increase  its  size  relative  to  other  information  on  the 
display. 

b.  Solution  and  Discussion 

The  font  size  requirements  listed  in  JCM-2154  for  a  26  inch  viewing 
distance  indicate  that  a  minimum  font  of  13  point  bold  is  required.  To  enhance  readability 
at  this  distance,  all  text  is  bold.  This  eliminates  the  use  of  this  mechanism  to  highlight 
information. 

Due  to  the  information  density  that  is  necessary  to  place  all  required 
information  on  the  ECO  displays,  highlighting  methods  such  as  highlighting,  increased 
brightness,  small  font  size  changes  and  changing  font  color  will  likely  not  be  very  effective. 
For  a  highlighting  method  to  be  effective  on  a  very  densely  packed  display,  it  must  truly 
set  the  information  apart  from  the  rest  of  the  display.  This  is  best  done  with  a  large  font 
change  or  with  a  combination  of  reverse  video  and  color.  “Two  to  four  different  character 
types  used  in  consistent  ways  are  optimal.”  (Mayhew,  1992)  Also,  evidence  shows  that 
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reverse  video  is  a  very  powerful  search  que  and  is  useful  to  indicate  that  an  item  has  been 
selected  or  to  indicate  an  error  field  (Mayhew,  1992). 

6.  Labeling  Conventions 

a.  Alternative  Approaches 

There  are  many  ways  to  label  information  on  the  screen.  Labels  could  be 
placed  above  or  to  one  side  of  the  data.  The  labels  could  be  all  capital  letters  or 
highlighted  in  some  way.  However  the  labeling  is  to  be  accomplished,  “every  data  item  on 
a  screen  should  be  labelled  in  some  way.”  (Tullis,  1988) 
h.  Solution  and  Discussion 

Labels  in  the  ECO  display  will  account  for  a  lot  of  the  screen  space  used, 
therefore  some  consistent  method  should  be  implemented.  Such  a  scheme  follows: 

•  Labels  should  contain  both  upper  and  lower  case  letters.  Mixed  case  text  is 
easier  to  read  than  all  uppercase  text.  Studies  show  a  13%  increase  in  reading 
speed  for  mixed  case  text.  (Mayhew,  1992) 

•  If  many  labels  appear  in  a  column,  they  should  be  right  justified.  Left  justified 
labels  can  lead  to  too  much  space  between  labels  and  data  elements.  (Tullis, 

1988) 

•  Each  label  should  be  followed  by  a  colon.  This  provides  consistency  among 
labels  and  identifies  text  as  a  label  and  not  a  data  element. 

•  Labels  should  reflect  correct  user  terminology.  Tullis  (1988)  states  that,  based 
on  empirical  data,  familiar  data  formats,  concise  wording  and  abbreviations 
should  be  used.  More  commonly,  this  is  known  as  speaking  the  user’s  language 
(Nielsen,  1993). 

There  might  be  times  when  such  a  labelling  scheme  cannot  be  followed. 

This  is  a  definite  problem  when  dealing  with  large  amounts  of  data  in  extremely  small 
screen  space,  such  as  in  the  ECO  displays.  In  such  cases,  as  many  of  the  above  guidelines 
as  possible  should  be  used. 
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7.  Displaying  Information  That  Will  Prevent  Launch 

a.  Alternative  Approaches 

Any  data  that  prevents  launching  a  missile  should  be  brought  to  the 
attention  of  the  ECO.  This  requires  that  the  information  be  highlighted  in  such  a  way  that 
it  is  easily  picked  out  from  other  data  on  the  display.  As  discussed  above,  options  include 
reverse  video  and  color  or  increased  font  size. 

b.  Solution  and  Discussion 

Increased  font  size  is  not  a  good  choice  for  highlighting  information  in  this 
case.  Because  of  the  density  of  the  information,  labels  and  information  must  be  spaced 
close  together.  Increasing  the  font  size  of  a  piece  of  information  may  cause  it  to  interfere 
with  nearby  data. 

“Color  is  very  effective  in  drawing  attention.”  (Mayhew,  1992)  It  can  also 
be  used  to  convey  status  to  help  the  user  determine  the  nature  of  the  message  even  before 
reading  it  (Mayhew,  1992).  Although  simply  changing  the  color  of  the  text  would  be 
effective,  redundant  coding  enhances  performance  (Mayhew,  1992).  Thus,  reverse  video 
should  be  used  to  provide  redundant  coding.  A  combination  of  reverse  video  and  color  is 
a  good  choice  to  highlight  information  that  will  prevent  launch  because  it  provides  a 
redundant  way  to  draw  attention  to  this  important  information. 

8.  Displaying  Missile  Progress 

a.  Alternative  Approaches 

There  are  two  basic  options  when  deciding  how  to  display  missile 
alignment  status:  textual  or  graphical.  The  process  proceeds  in  eight  modes,  each  having 
their  own  name.  Previous  versions  of  TWCS  used  a  textual  representation  that  listed  the 
missile’s  alignment  mode. 

b.  Solution  and  Discussion 

Tullis  (1988)  showed  that  initially,  graphical  representations  of  simple 
systems,  such  as  the  missile  alignment  status,  allowed  for  faster  interpretation  that 
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narrative  formats.  However,  as  users  became  more  used  to  both  types  of  display,  there 
was  no  difference  in  interpretation  times.  As  the  ECO  displays  should  be  usable  at  first 
glance  a  graphical  display  is  the  better  choice.  Also,  a  progress  bar  is  widely  becoming  the 
standard  metaphor  for  system  task  completion,  an  appropriate  widget  for  displaying 
missile  alignment  progress. 

9.  VLS  Layout 

0.  Alternative  Approaches 

A  full  size  VLS  launcher  is  has  eight  modules  of  eight  cells  each. 

Displaying  the  information  for  64  launcher  cells  in  a  very  limited  display  area  does  not 
allow  for  many  alternatives.  Any  method  that  is  chosen  should  accurately  represent  the 
physical  layout  of  the  VLS  launcher. 

b.  Solution  and  Discussion 

As  the  ECO  displays  are  being  added  to  an  existing  system,  the  best  option 
is  to  display  the  VLS  as  it  is  in  the  parent  system.  However,  the  limited  application  area 
of  the  ECO  displays  will  require  slight  modification  to  ensure  the  display  fits  in  the  display. 
The  only  significant  change  to  make  is  to  simply  reduce  the  amount  of  whitespace  in  the 
display. 

10.  Representing  Information  About  Cells  and  Modules  With  Faults 

a.  Alternative  Approaches 

The  informational  requirements  dictate  what  information  should  be 
contained  in  the  labels  for  each  launcher  cell.  Displaying  if  a  module  or  cell  has  a  fault  can 
be  done  by  highlighting  the  cell  or  module  in  some  way. 

b.  Solution  and  Discussion 

To  ensure  that  this  display  remains  consistent  with  the  rest  of  the  displays, 
the  method  used  to  display  a  cell/module  fault  should  be  the  same  as  the  method  used  to 
display  conditions  that  prevent  missile  launch.  The  label  for  any  cell  or  module  that 
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prevents  a  missile  launching  from  that  cell  or  module  will  be  reverse  video  with  a  red 
background  and  white  foreground. 

To  show  that  a  missile  has  been  selected,  a  more  subtle  form  of 
highlighting  is  required  to  keep  the  screen  from  appearing  too  cluttered.  A  form  of 
graphical  highlighting,  such  as  a  border  around  the  cell’s  information,  is  one  method  to  do 
this. 

11.  Plan  Status  Page  Layout 

a.  Alternative  Approaches 

The  informational  requirements  list  a  great  deal  of  information  that  is 
needed  for  each  active  plan.  It  is  obvious  that  if  several  plans  are  active  the  information 
required  to  be  displayed  will  quickly  fill  the  limited  application  area  of  the  ECO  display 
window.  Therefore,  some  method  of  paging  or  scrolling  through  the  list  of  active  plans  is 
required.  Two  possible  methods  of  accomplishing  this  are  push  buttons  or  scroll  bars. 
h.  Solution  and  Discussion 

To  display  the  required  data  for  each  plan,  several  informational  groupings 
should  be  used.  As  discussed  previously,  each  data  item  should  be  labeled.  Some  form  of 
separator  should  be  used  between  each  plan’s  status  to  aid  in  interpretation. 

The  most  consistent  method  to  view  the  large  numbers  of  active  plans  is  to 
use  a  scroll  bar.  The  data  should  be  displayed  within  an  X  workspace  and  scrolled  with 
the  associated  scroll  bar. 

F.  INITIAL  PROTOTYPE  DISPLAYS 

Figures  4  through  8  show  the  first  iteration  design  of  the  ECO  displays.  These 
initial  prototypes  will  undergo  heuristic  and  guideline  evaluation  to  discover  any  usability 
problems  prior  to  interactive  user  testing. 
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Figure  5.  Plan  Status  Display. 
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Figure  7.  Forward  VLS  Display. 
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IV.  HEURISTIC  AND  GUIDELINE  EVALUATION 


The  are  two  general  ways  to  test  a  user  interface:  with  and  without  users.  Both 
testing  with  users  and  testing  without  users  may  fail  to  reveal  all  the  problems  with  an 
interface.  A  combination  of  the  two  methods,  however,  significantly  improves  the  chances 
of  uncovering  ail  the  usability  problems  in  an  interface  (Lewis  and  Rieman,  1994).  No 
matter  which  methods  are  employed,  interface  testing  is  critical  to  the  success  of  an 
interface.  Not  only  is  there  a  cost  savings  associated  with  using  interface  testing  methods, 
but  there  is  increased  usability  as  well  (Nielsen,  1993).  This  increased  usability  is  apparent 
through  reduced  errors,  faster  user  response  times  and  shorter  user  training  times,  all  of 
which  are  essential  to  weapon  systems  (Nielsen,  1993). 

Testing  an  interface  with  users  can  be  expensive  (Nielsen,  1993).  It  requires 
access  to  a  group  of  test  users  that  are  as  representative  as  possible  of  the  intended  users 
of  the  system  (Nielsen,  1993).  Sometimes,  such  users  are  not  readily  available  or  have 
their  own  time  demands.  Because  of  this,  testing  an  interface  with  users  requires  a  large 
amount  of  coordination  between  the  testers  and  users  and  a  great  deal  of  prior  preparation 
(Lewis  and  Rieman,  1994).  Also,  an  interface  is  not  normally  ready  to  be  tested  by  live 
users  until  it  is  nearly  complete.  Changes  made  to  an  interface  at  this  stage  in  the  lifecycle 
can  be  very  costly  to  make  (Nielsen,  1993). 

From  a  time  and  cost  standpoint,  testing  an  interface  without  test  users  is  relatively 
less  inexpensive  as  it  does  not  require  advanced  planning  or  a  test  user’s  time  and  effort 
(Nielsen  and  Molich,  1990;  Lewis  and  Rieman,  1994).  Testing  an  interface  without  users 
can  be  conducted  early  in  the  usability  lifecycle  to  reveal  problems  that  may  hinder  user 
testing.  Problems  uncovered  at  this  stage  of  development  are  easier  to  correct  as  well 
(Nielsen,  1993). 

This  chapter  discusses  three  methods  of  testing  an  interface  without  users: 
heuristic  evaluation,  guideline  evaluation  and  cognitive  walkthroughs.  The  methods  are 
compared  to  other  forms  of  user  testing  and  two  methods  are  applied  to  the  initial 
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Engagement  Control  Officer  (ECO)  displays  presented  in  Figures  4  through  8.  The  results 
of  these  analyses  are  listed  and  the  resulting  second  iteration  ECO  displays  are  presented. 

A.  ANALYSIS  METHODS 

1.  Heuristic  Evaluation 

Heuristic  evaluation  is  an  informal  method  of  usability  analysis  where  evaluators 
use  their  experience  and  a  set  of  guidelines,  called  heuristics,  to  find  usability  problems 
with  an  interface  (Nielsen  and  Molich,  1990).  The  goal  of  heuristic  evaluation  is  to  find 
usability  problems  in  an  interface  so  they  can  be  resolved  as  part  of  the  iterative  design 
process  (Nielsen,  1993).  There  are  many  lists  of  heuristics  for  interface  evaluation.  They 
range  from  very  short,  very  general  guidelines  to  lists  containing  thousands  of  very 
specific  guidelines.  Both  approaches  have  their  problems.  Evaluations  using  very  general 
lists  ofl:en  miss  usability  problems  and  the  long  lists  are  usually  too  unwieldy  to  apply 
(Lewis  and  Rieman,  1994). 

a.  A  Set  of  Heuristics 

Nielsen  and  Molich  (1990)  introduce  a  set  of  nine  heuristics  for  to 
evaluating  the  usability  of  an  interface.  These  heuristics  were  developed  by  Nielsen  and 
Molich  based  on  their  experience  teaching  and  consulting  about  usability  engineering.  The 
nine  principles  are  generally  accepted  in  the  user  interface  community  and  are  present 
either  explicitly  or  implicitly  in  most  lists  of  principles  for  human-computer  interface  (HCI) 
design  (Lewis  and  Rieman,  1994).  Each  of  the  heuristics  is  discussed  briefly  below.  A 
more  thorough  discussion  of  each  heuristic  can  be  found  in  Nielsen  (1993). 

•  Simple  and  natural  dialog  -  User  interfaces  should  be  as  simple  as  possible, 
presenting  only  the  information  the  user  needs,  and  no  more,  exactly  when  it  is 
needed  (Nielsen,  1993).  This  information  should  be  in  an  order  that  matches  the 
user’s  task  at  hand  (Lewis  and  Rieman,  1994).  When  graphical  user  interfaces 
are  used,  these  concepts  can  be  extended  to  graphical  aspects  such  as  color  use 
and  graphical  boundaries. 

•  Speak  the  user’s  language  -  Since  an  interface  is  designed  for  a  user,  it  should 
use  terms  that  are  based  on  the  user’s  language.  When  users  have  their  own 
specialized  terminology  for  their  domain,  the  interface  should  use  it,  rather  than 


32 


more  commonly  used  everyday  language  (Nielsen,  1993).  System-specific 
engineering  terms  should  be  avoided.  This  concept  is  further  extended  to  the  use 
of  metaphors  in  the  interface.  Any  metaphors  that  are  used  should  be  correct 
mappings  from  the  user’s  conceptual  model  and  not  present  any  dual  meanings. 

•  Minimize  user  memory  load  -  The  user  should  not  be  made  to  remember 
information  given  by  the  interface.  Information  should  be  left  on  the  screen  until 
it  is  no  longer  needed  (Lewis  and  Rieman,  1994).  Also,  the  system  should  be 
built  around  a  small  number  of  rules  that  apply  throughout  the  interface  to  make 
it  easier  to  transition  from  one  part  to  the  next  without  having  to  relearn  or 
remember  such  rules  (Nielsen,  1993). 

•  Be  consistent  -  Consistency  applies  to  the  way  interface  elements  both  appear 
and  behave.  If  a  user  can  apply  an  action  in  one  situation  and  get  a  certain  result, 
they  should  expect  similar  results  when  applying  that  same  action  in  a  different 
situation.  This  behavioral  consistency  lets  users  feel  more  confident  when  using 
the  system  as  they  will  know  what  to  expect  when  repeating  the  same  action. 

The  same  information  should  be  formatted  in  the  same  way  on  every  screen  to 
facilitate  recognition.  (Nielsen,  1993)  This  consistency  in  appearance  can  help 
reduce  search  times  as  a  user  becomes  more  familiar  with  an  interface  and  knows 
where  to  expect  certain  information.  More  on  consistency  can  be  found  in 
Tognazzini  (1990). 

•  Provide  feedback  -  The  system  should  let  the  user  know  how  it  is  reacting  to 
their  input  at  all  times  (Nielsen,  1993).  Feedback  should  be  timely  and  expressed 
in  concrete  and  specific  terms.  Users  should  know  the  effects  their  actions 

are  having  on  the  system  (Lewis  and  Rieman,  1994). 

•  Provide  clearly  marked  exits  -  “The  system  should  offer  the  user  an  easy  way 
out  of  as  many  situations  as  possible.”  (Nielsen,  1993)  Exits  can  be  in  the  form 
of  Cancel  buttons,  escape  key  sequences  or  an  “undo”  facility.  However  they 
are  conveyed  to  the  user,  exits  should  be  obvious,  easily  accessed  and  quick. 
They  should  not  rely  on  the  user’s  memory  of  an  action  but  rather  be  labeled  and 
always  visible. 

•  Provide  shortcuts  -  Experienced  users  should  be  provided  with  quick  methods 
to  accomplish  tasks  without  having  to  view  information  they  do  not  need  (Lewis 
and  Rieman,  1994).  They  should  be  able  to  go  directly  to  a  desired 

location  in  the  interface  without  having  to  traverse  lengthy  menu  hierarchies. 

•  Good  error  messages  -  A  system  should  provide  error  messages  that  are  clearly 
worded,  precise,  unintimidating  and  offer  solutions  (Nielsen,  1993).  Systems 
should  also  provide  for  error  recovery  when  appropriate. 
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•  Prevent  errors  -  A  better  option  than  having  good  error  messages  is  to  avoid 
error  situations  in  the  first  place.  This  is  done  by  recognizing  error  prone 
situations  and  either  avoiding  them  entirely  or  providing  a  way  for  the  user  to 
proceed  through  the  situation  without  causing  an  error.  An  example  is  to 
provide  a  list  of  available  files  to  a  user  rather  than  have  them  type  in  a  filename 
that  might  be  invalid. 

b.  How  to  Apply  Heuristics 

Nielsen  and  Molich  (1990)  outlines  the  best  way  they  found  to  apply 
heuristics  to  evaluate  an  interface.  They  conducted  an  experiment  in  which  four  groups  of 
evaluators  heuristically  evaluated  separate  interfaces.  The  average  proportions  of  known 
usability  problems  found  were  51%,  38%,  26%  and  20%  in  the  four  experiments  (Nielsen 
and  Molich,  1990).  By  analyzing  the  results  of  their  experiment,  it  was  discovered  that  if 
the  usability  problems  found  by  individual  evaluators  were  combined  with  those  of  other 
evaluators  in  the  group,  the  overall  number  of  distinct  usability  problems  increased 
dramatically.  These  aggregates  of  evaluators  can  find  significantly  more  usability 
problems  as  the  number  of  evaluators  increases,  with  a  point  of  diminishing  returns  around 
the  point  often  evaluators  (Nielsen  and  Molich,  1990).  Table  1  shows  the  average 
proportion  of  usability  problems  found  in  each  of  the  four  interfaces  for  various  sized 
aggregates  of  evaluators. 


Number  of  Evaluators  in  Aggregate 

I 

2 

3 

5 

10 

Interface  1 

51% 

71% 

81% 

90% 

97% 

Interface  2 

38% 

52% 

60% 

70% 

83% 

Interface  3 

26% 

41% 

50% 

63% 

78% 

Interface  4 

20% 

33% 

42% 

55% 

71% 

Table  1.  Average  Proportion  of  Usability  Problems  Found  for  Various  Sized  Aggregates 
of  Evaluators.  After  Nielsen  and  Molich  (1990). 
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This  data  indicates  that  heuristic  evaluation  is  best  done  with  more  than 
one  person.  However,  even  with  only  one  evaluator,  a  fairly  large  number  of  problems  can 
be  found.  When  supplemented  with  another  usability  engineering  method  a  large  number 
of  problems  can  be  discovered  with  only  one  evaluator  (Nielsen  and  Molich,  1990). 
Another  way  to  increase  the  proportion  of  usability  problems  found  with  only  one 
evaluator  is  to  use  a  more  experienced  evaluator.  An  even  greater  number  of  problems 
can  be  discovered  if  the  evaluator  is  also  a  “double  expert,”  that  is,  the  evaluator  is 
experienced  in  the  interface’s  domain  as  well  as  in  interface  design  (Nielsen,  1993). 

2.  Guideline  Evaluation 

Guideline  evaluation  is  a  method  of  interface  evaluation  that  uses  very  specific 
recommendations  about  the  design  of  an  interface.  They  usually  include  such 
recommendations  as  how  the  contents  of  a  screen  should  be  organized  or  how  items 
should  be  arranged  in  a  menu.  The  number  of  guidelines  is  usually  very  large.  Guideline 
lists  are  normally  specific  enough  to  be  applied  by  people  unexperienced  in  user  interface 
evaluation  (Jeffries  et  al.  1991).  This  potentially  increases  the  number  of  people  that  are 
able  to  conduct  interface  evaluation  which  is  especially  helpful  when  usability  specialists 
are  unavailable. 

One  example  of  a  guideline  list  can  be  found  in  the  Open  Software  Foundation’s 
Motif  Style  Guide.  It  includes  an  80-page  Level  One  Certification  Checklist  that  specifies 
how  an  application  program  must  behave  to  be  certified  “OSF/Motif  Style  Guide 
Compliant”  (OSF,  1993).  Such  a  list  can  be  invaluable  to  an  interface  developer  to  ensure 
a  new  interface  complies  with  current  standards  and  to  point  out  possible  usability 
problems  associated  with  not  following  such  standards. 

3.  Cognitive  Walkthrough 

Cognitive  walkthroughs  are  a  method  of  interface  evaluation  in  which  interface 
developers  walk  through  the  interface  in  the  context  of  core  tasks  that  a  typical  user  needs 
to  accomplish  (Jeffries  et  al.  1991).  The  evaluation  proceeds  by  first  selecting  a  task  that 
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the  interface  was  designed  to  support.  The  evaluator  then  attempts  to  accomplish  the  task 
by  pretending  to  be  a  user  who  is  new  to  the  interface.  For  the  method  to  work  correctly, 
the  evaluator  must  make  good  assumptions  of  what  the  user  is  thinking.  While  carrying 
out  the  task,  if  the  evaluator  cannot  make  a  reasonable  assumption  based  on  his 
knowledge  of  the  user  and  the  prompts  and  information  supplied  by  the  interface,  a 
usability  problem  exists  (Lewis  and  Rieman,  1994). 

This  method  is  best  suited  for  a  task-centered  user  design  methodology  such  as 
that  described  in  Lewis  and  Rieman  (1994).  In  such  a  methodology,  the  interface  is 
designed  to  support  a  list  of  typical  tasks  which  are  specified  during  a  task  analysis. 
Walkthroughs  focus  most  clearly  on  problems  users  will  encounter  when  they  first  use  an 
interface  (Lewis  and  Rieman,  1994).  Also,  a  cognitive  walkthrough  is  really  a  tool  for 
developing  the  interface,  not  validating  it  (Lewis  and  Rieman,  1994). 

4.  A  Comparison  of  the  Methods 

Jeffries  et  al.  (1991)  experimentally  compares  four  methods  of  testing  an  interface: 
heuristic  evaluation,  guideline  evaluation,  cognitive  walkthroughs  and  usability  testing.  A 
single  interface  was  evaluated  by  four  teams,  each  using  one  of  the  above  methods. 

Specific  information  about  the  background  of  the  interface  reviewers  and  the  methods  they 
used  can  be  found  in  Jeffries  etal.  (1991). 

After  the  completion  of  each  evaluation,  the  errors  they  reported  were  analyzed. 
The  evaluations  uncovered  206  problems  that  addressed  usability  errors  which  were 
dubbed  “core”  problems.  Heuristic  evaluation  uncovered  105  core  problems  with  usability 
guidelines,  cognitive  walkthroughs  and  usability  testing  accounting  for  35,  35  and  31  of 
the  remaining  101  problems,  respectively  (Jeffries  c/ a/.,  1991). 

Each  core  problem  was  also  rated  by  the  testers  in  terms  of  severity  on  a  scale  of  1 
(trivial)  to  9  (critical).  These  ratings  took  into  account  the  impact  of  the  problem,  the 
frequency  it  occurred  and  the  number  of  users  that  it  would  affect.  (Jeffries  et  al,  1991) 
Using  these  ratings,  each  evaluation  method’s  mean  problem  severity  was  calculated. 
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Usability  testing  uncovered  the  most  severe  problems,  on  average,  with  a  mean  problem 
severity  of  4. 15.  Guidelines,  heuristic  evaluation  and  cognitive  walkthroughs  found 
problems  with  mean  severity  of  3.61,  3.59  and  3.44,  respectively  (Jeffries  etal,  1991). 

The  experimenters  further  analyzed  the  results  of  the  interface  evaluations  by 
calculating  a  benefit/cost  ratio.  The  benefit  was  found  by  summing  the  severity  scores  of 
all  the  core  problems  while  the  cost  is  simply  the  number  of  man-hours  spent  on  analysis. 
The  benefit/cost  ratio  was  determined  using  two  methods.  In  the  first,  the  time  that 
evaluators  using  the  guidelines  and  cognitive  walkthrough  methods  spent  becoming 
acquainted  with  the  interface  was  included.  The  second  ratio  does  not  take  into  account 
this  time  because  in  a  real  world  situation  the  interface’s  developers  would  be  conducting 
these  evaluations,  therefore,  this  additional  time  would  not  be  expended.  Table  2 
summarizes  the  results  of  the  benefit  cost  analysis  using  the  second  method. 


Heuristic 

Evaluation 

Usability 

Testing 

Guidelines 

Cognitive 

Walkthrough 

Sum  of 
severity  scores 

433 

133 

130 

120 

Analysis  time 
(man-hours) 

35 

199 

22 

37 

Severity/time 

12 

1 

6 

3 

Table  2.  Benefit/Cost  Ratios  for  the  Four  Evaluation  Techniques. 


After  Jeffries  eta/.,  1991. 


This  experiment  shows  that  techniques  for  testing  an  interface  without  users  are 
very  cost  effective,  especially  heuristic  evaluation.  However,  many  severe  problems 
simply  cannot  be  found  without  testing  an  interface  with  users  (Jeffries  etal,  1991)  .  It 
also  implies  that  a  combination  of  usability  testing  techniques  are  the  best  way  to  find  a 
wide  range  of  usability  problems. 
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B.  ANALYZING  THE  INTERFACE  WITHOUT  USERS 

1.  Evaluation  Methods  Used 

To  test  the  ECO  displays  without  users,  two  methods  are  employed:  heuristic 
evaluation  and  guideline  evaluation.  These  methods  are  used  because  of  their  high 
benefit/cost  ratio  and  their  ease  of  application.  The  combination  of  these  methods  can 
uncover  a  significant  number  of  usability  problems  as  shown  by  Jeffries  et  al.  (1991). 

The  heuristic  evaluation  of  the  ECO  displays  is  conducted  by  one  person  using  the 
heuristics  described  in  Nielsen  (1993).  Although  the  optimal  way  to  conduct  a  heuristic 
evaluation  is  to  use  aggregates  of  evaluators,  a  single  evaluator  can  uncover  a  significant 
number  of  usability  problems  (Nielsen  and  Molich,  1990).  The  results  of  a  heuristic 
evaluation  conducted  by  a  single  evaluator  should  not  be  the  sole  method  for  determining 
usability  problems  because,  as  shown  in  Nielsen  and  Molich  (1990),  a  single  evaluator 
can  miss  some  usability  problems.  In  order  to  uncover  the  maximum  percentage  of 
usability  problems,  single  evaluator  heuristic  evaluation  should  be  only  one  part  of  a 
usability  testing  plan. 

The  guidelines  used  to  conduct  the  guideline  evaluation  are  contained  in  Appendix 
B  of  JCM-2154.  This  46-page  checklist  ensures  that  a  user  interface  conforms  to  JCM- 
2154  standards.  Each  guideline  represents  a  single  HCI  requirement  from  JCM-2154. 

2.  Results  of  Heuristic  Evaluation 

Heuristic  evaluation  of  the  ECO  displays  reveals  nine  usability  problems.  The 
small  number  of  problems  can  be  attributed  to  two  reasons.  First,  the  evaluation  is 
conducted  by  only  one  evaluator  and,  for  the  reasons  discussed  previously,  one  cannot 
expect  this  method  to  uncover  a  large  proportion  of  usability  problems  (Nielsen  and 
Molich,  1990).  Second,  the  ECO  displays  are  veiy  simple  in  terms  of  user  interaction. 

The  displays  consist  of  only  five  display  pages,  only  three  of  which  have  interface  elements 
that  can  interact  with  the  user.  Table  3  describes  each  problem  and  its  resolution. 
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Heuristic 

Problem  Description 

Problem  Resolution 

ECO  Summary 

Speak  the  user’s 
language 

Several  labels  are  abbreviated 
incorrectly. 

Change  labels  to  more 
acceptable  abbreviations. 

EGO  Summary 

Be  consistent 

Position  of  information  given  for 
next  launch  is  not  consistent 
with  the  same  data  listed  on  the 
Plan  Status  Display. 

Change  information’s  position 
to  the  same  as  listed  tor  each 
plan  on  the  Plan  Status 

Display 

ECO  Summary 

Be  consistent 

Information  about  Salvo  is  not 
consistent  with  the  information 
given  on  the  Plan  Status 

Display. 

Change  information  given  to 
“Salvo/RS/BU”  to  become 
consistent  with  Plan  Status 
Display. 

ECO  Summary 

Be  consistent 

Labels  of  groups  are  not 
consistent.  ‘‘VLS  Status”  label 
is  not  centered  over  the  group  it 
labelled. 

Move  label  to  center  of  group. 

ECO  Summary 

Be  consistent 

Labels  of  conditions  signifying  a 
launch  is  not  possible  are  not 
consistent  in  their  use  of 
capitalization. 

Change  labels  of  all 
conditions  that  prevent  launch 
to  contain  all  uppercase  letters 
only. 

Plan  Status 

Simple  and 
natural  dialog 

Violates  the  gesalt  rules  for 
human  perception  as  described 
in  Nielsen  (1993).  This 
decreases  the  user’s  ability  to 
perceive  relationships  between 
interface  elements. 

Modify  arrangement  of 
information  to  improve  group 
perception. 

Plan  Status 

Be  consistent 

Method  of  setting  off  groups  of 
data  is  inconsistent  with  that 
used  on  ECO  Summary  Display 
and  COI/CCOI  Display. 

Change  grouping  method  to 
that  used  on  ECO  Summary 
Display  and  COI/CCOI 

Display 

Plan  Status 

Be  consistent 

! 

Method  of  labelling  groups  is 
inconsistent  with  that  used  on 

ECO  Summary  Display  and 
COI/CCOI  Display. 

Move  group  labels  to  a 
position  centered  above  each 
group. 

COI/CCOIs 

Speak  the  user’s 
language 

Abbreviation  for  course,  “Crs”, 
is  not  the  normal  abbreviation 
used. 

Change  label  to  “Cse,”  a  more 
j  standard  abbreviation. 

Table  3.  Heuristic  Evaluation  Results. 
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3.  Results  of  Guideline  Evaluation 

Using  the  guideline  checklist  in  JCM-2154,  seven  guideline  violations  can  be  found 
in  the  ECO  displays.  Table  4  shows  which  guidelines  are  violated  and  the  action  required 
to  correct  the  problem. 


I^E^^SSSiSII 

JCM-2154  Guideline 

Corrective  Action 

COI/CCOIs 

When  applications  provide  the  capability  to  sort 
the  contents  of  a  multi-column  list  box  the 
heading  of  each  column  shall  seem  to  be  a  push 
button. 

Remove  “Sort  by”  option  menu. 
Replace  column  labels  with  push 
buttons  whose  function  is  to  sort 
the  list  by  that  column  when 
pressed. 

Forward  and  Aft 
VLS  Status 

A  push  button  shall  be  used  to  initiate  action. 

Change  the  representation  of  each 
cell  from  a  push  button  to  a 
graphic  box. 

All 

When  a  widget  contains  an  error,  its  background 
shall  remain  IndianRed2. 

Although  this  does  not  apply 
directly,  to  remain  consistent  with 
JCM-21 54,  change  background 
color  of  conditions  preventing 
launch  to  IndianRed2. 

All 

All  text  shall  be  of  the  Helvetica  font. 

Change  font  to  Helvetica. 

All 

1  The  title  shall  be  centered  in  the  title  bar  and 
presented  in  uppercase  letters. 

Change  titles  to  uppercase  letters. 

Plan  Status 

Data  groupings  shall  be  indicated  with  blank 
space,  separator  lines,  and/or  different  intensity. 

Revise  method  of  grouping 
information. 

COI/CCOIs 

If  the  window  contains  many  rows  and  columns, 
a  blank  line  shall  be  inserted  after  every  third  to 
fifth  row  and  three  spaces  inserted  between 
every  column. 

Add  blank  line  in  between  every 
third  data  set. 

Table  4.  Guideline  Evaluation  Results. 


4.  Effectiveness  of  Heuristic  and  Guideline  Evaluation 

Because  one  cannot  know  in  advance  the  number  of  usability  problems  an  interface 
contains,  it  is  unclear  what  proportion  of  the  total  problems  heuristic  and  guideline 
evaluation  uncovers.  Further  evaluation  by  additional  or  more  experienced  evaluators  may 
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reveal  a  larger  number  of  usability  problems  (Nielsen  and  Molich,  1990).  Changes  made 
to  the  ECO  displays  based  on  the  results  of  the  heuristic  and  guideline  evaluation  will 
make  the  interface  more  consistent  with  both  accepted  HCI  standards  and  JCM-2154 
requirements  (Lewis  and  Rieman,  1994).  This  will  reduce  chances  of  user  error,  increase 
the  speed  and  accuracy  of  data  assimilation  and  reduce  training  time  for  the  interface 
(Nielsen,  1993). 

C.  SECOND  ITERATION  DISPLAYS 

Figures  9  through  1 3  show  the  second  iteration  of  the  ECO  displays.  These  will  be 
subjected  to  user  testing  to  reveal  any  further  usability  problems  in  the  interface. 
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Figure  10.  Second  Iteration  Plan  Status  Display 
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V.  USABILITY  TESTING 


“You  can’t  really  tell  how  good  or  bad  your  user  interface  is  going  to  be  without 
getting  people  to  use  it.”  (Lewis  and  Rieman,  1994)  Testing  an  interface  with  users  is  the 
most  fundamental  usability  testing  method  because  it  provides  the  opportunity  to  find 
usability  problems  that  can  be  missed  by  evaluation  methods  such  as  heuristic  evaluation, 
cognitive  walkthrough  and  guideline  evaluation  (Nielsen,  1993),  Many  of  the  most  severe 
problems  in  a  user  interface  simply  cannot  be  found  without  using  real  users  to  test  the 
interface  (Jeffries  et  a/. ,  1 99 1 ). 

This  chapter  discusses  some  issues  involved  in  usability  testing.  One  method  of 
usability  testing,  thinking  aloud,  is  explored  in  detailed  and  the  results  of  using  this 
method  on  the  Engagement  Control  Officer  (ECO)  displays  is  presented.  Usability 
problems  uncovered  by  user  testing  are  then  corrected  and  a  set  of  third  iteration  displays 
is  presented. 

A.  ISSUES  IN  USABILITY  TESTING 

1.  Interface  Test  Users 

Choosing  users  to  test  an  interface  can  be  as  important  as  choosing  a  usability 
method  to  use.  Test  users  should  be  as  representative  of  the  intended  users  of  the  system 
as  possible  (Nielsen,  1993).  If  the  exact  individuals  for  whom  a  system  is  being  designed 
can  be  identified,  then  they  should  be  used.  If  such  users  cannot  be  identified,  subjects 
from  closely  related  user  populations  might  be  acceptable  subjects.  For  example,  if  a 
system  is  being  designed  for  doctors,  medical  students  may  be  good  test  users  with  more 
available  time  than  practicing  physicians  (Lewis  and  Rieman,  1994). 

Another  issue  when  selecting  test  users  is  to  account  for  both  novice  and  expert 
users.  Because  of  the  disparity  in  their  knowledge,  novice  and  expert  users  should  be 
tested  with  separate  tests.  These  tests  should  contain  common  elements  but  also  include 
different  test  tasks,  (Nielsen,  1993) 


47 


There  are  also  many  ethical  concerns  when  dealing  with  test  users.  Users  must  be 
made  to  feel  comfortable  and  their  emotions  and  well-being  should  always  be  kept  in  mind 
(Nielsen,  1993),  Test  users  often  feel  a  great  deal  of  pressure  to  perform.  Many  times,  a 
usability  problem  with  an  interface  will  cause  a  user  to  worry  that  they  lack  the  ability  or 
intelligence  to  use  the  system  correctly.  Users  should  be  made  to  understand  that  it  is  the 
system  that  is  being  tested,  not  them.  They  should  also  be  aware  that  their  inability  to 
perform  a  certain  task  is  not  a  reflection  on  their  ability,  but  rather  the  inability  of  the 
interface  to  provide  them  the  means  to  accomplish  it.  (Lewis  and  Rieman,  1994)  More 
on  the  ethical  aspects  of  user  testing  can  be  found  in  Nielsen  (1993)  and  Lewis  and 
Rieman  (1994). 

2.  Types  of  Data 

Lewis  and  Rieman  classify  the  data  that  can  be  collected  during  usability  testing  as 
either  “process”  data  or  “bottom-line”  data.  Process  data  are  observations  of  what  a  test 
user  is  doing  during  the  test  step-by-step  and,  hopefully,  why  they  are  doing  it.  Bottom- 
line  data  are  typically  quantitative  measurements  taken  during  a  usability  test,  such  as  the 
time  a  user  takes  to  complete  a  task.  (Lewis  and  Rieman,  1994) 

Collection  of  process  data  is  also  know  as  formative  evaluation.  The  main  goal  of 
a  test  using  this  form  of  data  collection  is  to  determine  which  aspects  of  a  user  interface 
are  good  or  bad  and  to  determine  what  changes  should  be  made  to  a  design.  Formative 
evaluation  is  often  done  as  part  of  the  iterative  design  process  and  can  be  accomplished 
with  only  a  few  users.  One  usability  testing  method  to  use  for  formative  evaluation  is  the 
thinking  aloud  test,  discussed  below.  (Nielsen,  1993) 

Summative  evaluation  is  an  attempt  to  assess  the  quality  of  an  interface  by 
collecting  and  analyzing  bottom-line  data.  This  form  of  evaluation  is  often  done  to 
compare  two  alternative  interfaces  or  as  a  method  to  perform  a  competitive  analysis. 
Measurement  tests  are  typically  used  to  perform  a  summative  evaluation.  In  measurement 
tests,  an  aspect  of  the  interface  is  given  a  quantifiable  measurement  that  will  be  collected 
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during  the  usability  test.  Nielsen  (1993)  lists  several  quantifiable  usability  measurements. 
This  bottom-line  data  is  recorded  and  analyzed  to  determine  how  well  an  interface  meets 
expected  usability  goals  or  how  one  interface  compares  to  another.  The  collection  of 
bottom-line  data  takes  a  large  number  of  user  to  ensure  the  data  is  both  valid  and  reliable. 
(Nielsen,  1993) 

3.  Deciding  on  a  Usability  Method 

Choosing  a  usability  method  can  depend  on  many  factors.  But  no  matter  which 
method  is  chosen,  relying  on  the  results  of  a  single  usability  method  to  the  exclusion  of 
others  is  not  recommended.  Many  usability  methods  can  supplement  one  another  because 
they  address  different  parts  of  the  usability  lifecycle  and  the  strengths  of  one  method  can 
make  up  for  the  weaknesses  of  another.  (Nielsen,  1993) 

Possibly  the  biggest  factor  that  affects  the  choice  of  usability  method  is  the  goal  of 
the  test.  One  would  not  choose  a  qualitative  or  subjective  test,  such  as  thinking  aloud,  to 
validate  a  system  because  the  opinions  of  a  few  test  users  may  not  reflect  those  of  the 
actual  target  user  group.  Another  factor  that  is  closely  related  to  the  test  goal  is  the  stage 
of  the  usability  lifecycle  in  which  the  test  is  conducted.  Many  methods,  such  as  hueristic 
analysis,  are  best  done  early  in  the  lifecycle  while  others,  such  as  performance 
measurements,  are  more  appropriate  to  the  end  of  the  lifecycle. 

Another  factor  that  drives  the  choice  of  which  usability  method  to  use  is  the 
number  of  test  users  available.  When  few  users  are  available,  it  is  best  to  concentrate  on 
methods  such  as  hueristic  analysis  and  thinking  aloud.  Methods  such  as  performance 
measurement  require  a  large  number  of  users  in  order  to  gather  enough  bottom-line  data 
for  the  measurements  to  be  considered  valid  and  reliable.  (Nielsen,  1993) 

Table  5  summarizes  the  advantages  and  disadvantages  of  several  usability  methods. 
The  table  is  necessarily  simplified;  however,  Nielsen  (1993)  covers  all  the  usability 
methods  listed  in  detail.  From  Table  5  one  can  determine  which  usability  method  is  best 
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suited  to  a  particular  situation  based  on  the  usability  lifecycle  stage  and  number  of  test 


users  available. 


Method 

Name 

Lifecycle  Stage 

Users 

Needed 

Main  Advantage 

Main  Disadvantage 

Hueristic 

evaluation 

Early  design,  “inner 
cycle”  of  iterative 
design 

None 

Finds  individual 
usability  problems.  Can 
address  expert  user 
issues. 

Does  not  involve  real 
users,  so  does  not  find 
“surprises”  relating  to 
their  needs. 

Performance 

measurements 

Competitive 
analysis,  final 
testing 

At  least 

10 

Hard  numbers.  Results 
easy  to  compare. 

Does  not  find  individual 
usability  problems. 

Thinking  aloud 

Iterative  design, 

formative 

evaluation 

3-5 

Pinpoints  user 
misconceptions.  Cheap 
test. 

Unnatural  for  users. 

Hard  for  expert  users  to 
verbalize. 

Observation 

Task  analysis, 
follow-up  studies 

3  or  more 

Ecological  validity; 
reveals  users’  real  tasks. 
Suggests  functions  and 
features. 

Appointments  hard  to 
set  up.  No  experimenter 
control. 

Questionnaires 

Task  analysis, 
follow-up  studies 

At  least 

30 

Finds  subjective  user 
preferences.  Easy  to 
repeat. 

Pilot  work  needed  (to 
prevent  misunderstand¬ 
ings). 

Interviews 

Task  analysis 

5 

Flexible,  in-depth 
attitude  and  experience 
probing. 

Time  consuming.  Hard 
to  analyze  and  compare. 

Focus  groups 

Task  analysis,  user 
involvement 

6-9  per 
group 

Spontaneous  reactions 
and  group  dynamics. 

Hard  to  analyze.  Low 
validity. 

Logging  actual 
use 

Final  testing, 
follow-up  studies 

At  least 

20 

Finds  highly  used  (or 
unused)  features.  Can 
run  continuously. 

Analysis  programs 
needed  for  huge  mass  of 
data.  Violations  of 
users’  privacy. 

User  feedback 

Follow-up  studies 

Hundreds 

Tracks  changes  in  user 
requirements  and  views. 

Special  organization 
needed  to  handle 
replies. 

Table  5.  Summary  of  Usability  Methods.  From  Nielsen  (1993). 
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B.  THINKING  ALOUD 


A  combination  of  usability  methods  that  is  often  useful  is  heuristic  evaluation  and 
thinking  aloud.  First  a  hueristic  evaluation  is  conducted  to  clean  up  the  interface  and 
correct  any  “obvious”  usability  problems.  After  redesign,  the  interface  is  subjected  to  user 
testing  using  the  thinking  aloud  method  to  check  the  outcome  of  the  iterative  design  step 
and  to  find  remaining  problems  that  were  not  discovered  by  the  hueristic  evaluation. 
(Nielsen,  1993) 

There  are  two  major  reasons  for  alternating  between  heuristic  evaluation 
and  user  testing  as  suggested  here.  First,  a  heuristic  evaluation  pass  can 
eliminate  a  number  of  usability  problems  without  the  need  to  “waste  users,” 
who  sometimes  can  be  difficult  to  find  and  schedule  in  large  numbers. 

Second,  these  two  categories  of  usability  assessment  methods  have  been 
shown  to  find  fairly  distinct  sets  if  usability  problems,  meaning  that  they 
supplement  each  other  rather  than  leading  to  repetitive  findings.  (Nielsen, 

1993) 

'"Thinking  aloud  may  be  the  single  most  valuable  usability  engineering  method.” 
(Nielsen,  1993)  The  idea  behind  thinking  aloud  is  simple.  A  user  is  asked  to  complete  a 
task  and,  while  they  work  on  it,  they  are  to  express  their  thoughts  verbally  (Lewis  and 
Rieman,  1994).  This  usability  method  allows  testers  to  understand  how  the  user  views  an 
interface  making  it  easy  to  identify  any  major  misconceptions  the  user  might  have 
(Nielsen,  1993).  The  thinking  aloud  method  has  the  advantage  of  being  able  to  elicit  a 
large  amount  of  qualitative  information  fi'om  a  small  number  of  users.  Its  disadvantage  is 
that  it  does  not  lend  itself  to  any  types  of  performance  measurement.  (Nielsen,  1993) 
Lewis  and  Rieman  (1994)  describes  many  of  the  aspects  of  the  thinking  aloud 
procedure.  These  aspects  are  summarized  in  the  following  list: 

•  Instructions  -  The  user  should  understand  that  the  tester  wishes  to  hear  what 
they  are  thinking  about  while  performing  a  task.  They  should  understand  that 
the  interface  is  being  tested,  not  them.  (Lewis  and  Rieman,  1994) 

•  The  observer’s  role  -  The  observer  should  remain  with  the  user  to  provide 
assistance  if  necessary.  Users  will  not  normally  give  a  steady  flow  of  comments 
unless  prompted  occasionally  (Lewis  and  Rieman,  1994).  The  observer  may 
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have  to  ask  questions  such  as  “What  do  you  think  that  means?” 

when  a  user  becomes  confiised  to  elicit  constructive  remarks.  (Nielsen,  1993) 

•  Recording  -  Information  can  be  recorded  by  taking  notes  on  paper  or  may  be 
accomplished  by  videotaping  the  test  user.  No  matter  how  it  is  done,  the  goal  is 
to  record  the  users  thoughts  while  performing  an  action. 

•  Summarizing  the  data  -  A  list  of  all  the  problems  the  user  had  should  be  made.  If 
possible,  the  reason  why  each  difficulty  arose  should  be  added  to  the  list.  (Lewis 
andRieman,  1994) 

•  Using  the  results  -  The  results  should  be  looked  at  from  two  aspects.  First,  does 
the  data  show  that  the  interface  worked  as  intended  or  did  the  users  take 
different  approaches  than  expected?  Second,  based  on  the  user’s  difficulties, 
how  important  and  difficult  is  each  problem  to  correct? 

C.  USER  TESTING  AND  RESULTS 

1.  Test  Plan 

User  testing  is  carried  out  using  the  thinking  aloud  method.  This  method  is  well 
suited  to  test  the  ECO  displays  as  they  are  still  in  the  iterative  design  portion  of  the 
lifecycle  and  few  test  users  are  available.  Users  are  instructed  to  describe  their  thought 
process  while  doing  so.  An  observer  is  present  to  both  record  users’  comments  and  assist 
the  users  only  when  necessary.  The  resulting  data  is  summarized  and  subsequently  used  to 
make  corrections  to  the  ECO  displays. 

2.  User  Sketch 

Three  test  users  are  all  U  S.  Navy  officers  with  experience  on  board  Tomahawk 
Land  Attack  Missile  (TLAM)  capable  ships.  All  three  users  are  qualified  ECOs  and  one  of 
the  users  has  experience  as  Strike  Warfare  Officer,  whose  primary  job  is  the  maintenance 
and  employment  of  the  Tomahawk  cruise  missiles  and  their  support  systems.  These  test 
users  are  very  representative  of  the  users  who  would  use  the  ECO  displays  during  a 
TLAM  strike. 

3.  Problems  Found  and  Resolution 

User  testing  using  the  thinking  aloud  method  with  three  test  users  reveals  eight 
additional  usability  problems.  Each  problem,  the  reason  why  it  occurred  and  its  resolution 
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is  listed  in  Table  6.  The  usability  problems  found  in  user  testing  are  the  basis  for 
corrections  to  the  second  iteration  prototype  displays. 


lEUBlIIIII 

Usability  Problem 

Reason  for  Problem 

Resolution 

All 

The  label  for  the  nearest 

CCOI  is  confusing.  Users 
are  unable  to  determine  what 
the  label  or  information 
meant. 

The  label  does  not  convey 
the  proper  information.  In 
an  attempt  to  save  space,  the 
label  was  shortened  to  be  the 
track  number  of  the  CCOI. 
Users  are  unable  to  make  the 
connection. 

Debriefing  with  users 
found  they  do  not  feel  the 
information  is  critical. 

The  best  solution  is  to 
remove  the  label. 

All 

The  label  for  OTCIXS  time 
late  is  misunderstood. 

Improper  terminology  causes 
misinterpretation  by  the  test 
users. 

Change  the  label  to  “Last 
Data:” 

All 

Label  for  system  status 
(Training/Tactical)  is  not 
obvious  enough. 

The  label’s  position  in  the 
middle  of  the  status  bar 
keeps  it  from  standing  out. 

Move  system  status  to  the 
left  side  of  the  status  bar 
to  make  it  more 
prominent. 

Forward 
Vertical 
Launching 
System  (VLS) 
Status  and  Ait 
VLS  Status 

Users  are  unable  to 
determine  which  half 
modules  are  unavailable  due 
to  either  a  missile  already 
being  selected  or  a  module 
fault. 

Missiles  that  are  unavailable 
for  selection  are  not  specified 
in  any  way. 

Reduce  the  intensity  of 
missiles  that  are 
unavailable  for  selection . 

ECO 

Summary  and 
Plan  Status 

Label  for  Salvo/RS/BU  is 
misunderstood. 

The  “BU”  portion  of  the 
label  is  not  needed  and 
causes  confusion.  A  “back¬ 
up”  mission  is  an  operational 
concept  and  not  something 
specified  in  the  weapon 
system. 

Remove  “/BU”  from 
label. 

ECO 

Summary 

Information  for  Controlling 
Weapon  Control  System 
(WCS)  causes  confusion. 

The  information  does  not 
represent  the  true  state  of  the 
WCS  switch. 

Change  information  for 
Controlling  WCS  to  read 
Off,  Manual  or  Auto. 

ECO 

Summary 

Label  for  Inertial  Navigation 
System  (INS)  causes 
confusion. 

The  INS  is  normally  referred 
to  by  its  equipment 
designator  WSN. 

Change  label  to  read 

WSN  instead  of  INS. 

Table  6.  Usability  Testing  Results. 
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Display 

Usability  Problem 

Reason  for  Problem 

Resolution 

OSV  Hierarchy 

To  change  the  page  to 
one  VLS  status  page 
while  viewing  the  other 
requires  too  many 
keystrokes.  Users 
prefer  to  switch  back 
and  forth  between  these 
two  pages  easily. 

After  selecting  a  VLS 
status  page,  the  OS  Vs 
return  to  their  original 
state.  To  switch  to  the 
other  VLS  status  page 
requires  selecting  the 
‘VLS  Status’  button 
again  and  then  pressing 
the  OSV  for  the  desired 
VLS  status  page. 

When  the  user  selects  a 
VLS  status  page  for 
display,  the  VLS  Status 
OSV  should  be 
relabeled  to  the  other 

VLS  status  page.  For 
example,  when  viewing 
the  Forward  VLS  Status 
page,  the  OSV  would 
read  “Aft  VLS  Status.” 
This  revised  hierarchy 
enables  fast  switching 
between  VLS  status 
pages. 

(Table  6  continued.)  Usability  Testing  Results. 


D.  THIRD  ITERATION  DISPLAYS 

The  displays  in  Figures  14-18  are  the  third  iteration  of  the  ECO  displays.  These 
prototype  displays  can  be  used  to  conduct  a  summative  evaluation  to  determine  the 
effectiveness  of  the  interface  design. 
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Figure  14.  Third  Iteration  ECO  Summary  Display 
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Figure  16.  Third  Iteration  COI/CCOI  Display. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Iterative  interface  design  is  an  excellent  method  to  produce  a  user  interface  that  is 
focused  on  the  needs  of  the  user.  Throughout  the  usability  lifecycle,  the  interface  design 
should  be  evaluated  using  an  appropriate  usability  method.  This  ensures  that  usability 
problems  are  found  as  early  as  possible  in  the  design  process.  This,  in  turn,  enables 
usability  problems  to  be  corrected  when  they  are  easiest,  and  cheapest,  to  fix. 

Assessing  the  usability  of  an  interface  does  not  necessarily  require  large  numbers 
of  test  users  or  expensive  usability  testing  laboratories.  Methods  such  as  heuristic  and 
guideline  evaluation  provide  a  means  to  effectively  evaluate  an  interface  without  the  need 
for  any  users.  They  also  have  the  advantage  of  requiring  little  preparation,  making  them 
easy  to  apply.  The  main  disadvantage  of  evaluating  an  interface  without  users  is  that  they 
are  simply  unable  to  replace  the  insights  that  a  real  user  can  provide.  An  excellent  method 
of  testing  with  real  users  is  thinking  aloud.  This  method  of  user  testing  reveals  what  a 
user  is  think  about  while  using  an  interface.  These  insights  are  invaluable  for  correcting 
user  misconceptions. 

The  Engagement  Control  Officer  (ECO)  displays  are  the  result  of  applying  an 
iterative  design  process.  By  employing  usability  methods  early  in  the  iterative  process, 
usability  problems  in  the  displays  are  corrected  before  they  become  costly  to  fix.  Testing 
the  ECO  displays  with  test  user  that  are  representative  of  the  intended  user  group  ensures 
that  the  displays  meet  user  requirements  and  agree  with  user  perceptions. 

The  ECO  displays  are  a  valuable  addition  to  the  Advanced  Tomahawk  Weapon 
Control  System  (ATWCS).  The  information  that  they  present  is  necessary  for  the  ECO  to 
complete  the  mission  of  launching  a  successful  Tomahawk  Land  Attack  Missile  (TEAM) 
strike.  The  system  state  and  weapon  status  are  accurately  depicted  to  aid  the  ECO  in 
making  correct  and  timely  decisions.  By  using  sound  screen  design  principles,  the  displays 
present  the  information  in  such  a  way  that  it  can  be  read  accurately  and  rapidly  by  the 
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ECO.  Usability  testing,  both  with  and  without  test  users,  shows  that  the  displays  are 
consistent,  use  proper  terminology  and  do  not  present  information  in  a  way  that  it  could 
easily  be  misinterpreted. 

B.  RECOMMENDATIONS 

The  following  list  of  recommendations  are  suggestions  to  improve  the  usability  of 
the  ECO  displays.  They  also  include  recommendations  to  improve  the  usability  of 
ATWCS  for  ECOs. 

•  Perform  additional  requirements  analysis  after  the  installation  of  ATWCS  in 
fleet  units  -  User  perceptions  and  expectations  may  change  after  they  become 
familiar  with  the  capabilities  of  the  new  system.  Additional  requirements  analysis 
will  ensure  that  the  users’  needs  are  continuing  to  be  met.  A  new  requirements 
analysis  will  also  reveal  requirements  that  result  from  advances  in  Tomahawk 
Land  Attack  Missile  (TEAM)  technology. 

•  Increase  the  size  of  the  secondary  screen  -  The  extremely  small  size  of  the 
secondary  screen  severely  restricts  the  design  of  displays  designated  for  this 
screen.  A  larger  screen  would  increase  the  possible  design  alternative  available 
for  displays  such  as  the  ECO  displays. 

•  Add  remote  displays  to  the  hardware  configuration  of  ATWCS  -  The  nature  of 
the  ECO’s  responsibilities  require  that  they  remain  standing  during  a  TEAM 
strike,  preventing  them  from  sitting  at  a  stationary  console.  Remote  displays  in 
addition  to  the  secondary  screen  would  allow  the  ECO  to  keep  appraised  of  a 
strike’s  status  without  having  to  continually  return  to  the  ATWCS  consoles. 
Along  with  this,  the  ECO  could  be  provided  some  form  of  remote  control  to 
change  the  display  shown  on  the  remote  screens. 

C.  SUGGESTIONS  FOR  FURTHER  RESEARCH 

Below  is  a  list  of  areas  that  present  opportunities  for  further  research. 

•  Perform  heuristic  evaluation  on  the  ECO  displays  with  several  more  evaluators  - 
As  shown  in  Nielsen  and  Molich  (1990),  the  number  of  usability  problems  found 
by  heuristic  evaluation  increases  significantly  when  aggregates  of  evaluators  are 
used.  These  results  could  be  compared  to  those  found  by  a  single  evaluator  to 
improve  the  ECO  displays  and  further  validate  the  findings  of  Nielsen  and 
Molich  (1990). 

•  Perform  a  summative  evaluation  of  the  ECO  displays  -  Quantitative  tests  will 
uncover  fijither  usability  problems  and  aid  in  the  validation  of  the  interface 
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design.  Measurement  tests  could  also  be  used  to  compare  the  ECO  displays 
presented  here  with  alternative  designs, 

•  Perform  a  task  analysis  for  the  ECO  displays  -  A  task  analysis  would  3deld 
representative  task  that  would  need  to  be  accomplished  using  the  ECO  displays. 
These  tasks  could  be  used  to  further  refine  the  ECO  displays  and  to  perform 
additional  usability  testing  using  the  cognitive  walkthrough  method. 
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